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Abstract 

Promoting  competition  in  contracting  has  been  a  focus  of  Department  of  Defense  (DoD) 
efforts  for  many  years,  but  this  focus  has  heightened  in  recent  years  as  defense  budgets 
have  declined  dramatically.  Though  there  has  been  much  written  about  the  results  of  these 
efforts,  most  of  the  literature  has  focused  on  rates  of  competition  for  the  DoD  overall.  In 
previous  work  on  DoD  contracting,  however,  the  Center  for  Strategic  and  International 
Studies  (CSIS)  has  found  that  competition  rates  vary  not  just  on  what  is  being  purchased,  but 
by  who  in  the  DoD  is  doing  the  purchasing. 

This  research  effort  builds  upon  previous  CSIS  work  on  defense  contracting,  using  publicly 
available  data  from  the  Federal  Procurement  Data  System.  The  study  team  has  undertaken 
to  build  a  predictive  model  of  defense  contracting,  identifying  factors  that  correlate  with  higher 
or  lower  rates  of  effective  competition,  and  using  data  from  past  years  to  generate  an 
“expected”  rate  of  competition.  With  this  model,  CSIS  digs  a  step  deeper  by  examining  trends 
in  competition  by  Place  of  Performance  (states)  and  by  Major  Contracting  Command.  The 
findings  from  this  analysis  are  used  to  generate  an  improved  predictive  model,  which  is 
previewed  in  this  report 
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Introduction1 


The  Defense-Industrial  Initiatives  Group  at  the  Center  for  Strategic  and  International 
Studies  (CSIS)  has  completed  the  work  detailed  in  this  paper  as  an  intermediate  effort  for  a 
larger  study  being  conducted  for  the  Naval  Postgraduate  School.  The  following  paper  details 
the  structure  of  the  team’s  efforts  to  create  a  predictive  model  of  the  degree  of  effective 
competition  in  defense  contracting.  Our  team  defines  effective  competition  as  all  contracts 
that  solicit  two  or  more  bids.  CSIS  believes  that  effective  competition  is  the  best  metric  to 
measure  whether  the  DoD  is  successfully  promoting  competition,  as  it  excludes 
competitively-sourced  contracts  that  receive  only  one  offer.  The  final  product  of  this  research 
effort  will  delve  deeper  into  the  issue  of  single-offer  competition,  but  the  study  team  believes 
that  many  cases  of  single-offer  competition  never  had  a  chance  of  getting  more  than  one 
offer. 


The  purpose  of  the  model  described  in  this  paper  is  to  better  understand  the  extent 
to  which  different  factors  in  defense  contracting  influence  effective  competition  in  the  bidding 
process.  This  effort  is  an  important  addition  to  the  literature  on  defense  competition,  as  it 
quantifies  the  correlative  effects  of  different  factors  through  a  statistically  rigorous  analysis. 
Numerous  statements  from  officials  inside  the  DoD  and  throughout  the  federal  government 
have  emphasized  the  importance  of  promoting  competition,  particularly  in  a  difficult  budget 
environment,  as  a  way  to  improve  quality  and  reduce  cost. 

The  larger  study  analyzes  competition  via  a  range  of  approaches.  Earlier,  under  this 
research  effort,  the  study  team  reported  on  levels  of  effective  competition  for  the  military 
departments  and  then  disaggregated  the  effective  competition  rates  for  products,  services, 
and  R&D.2  This  approach  allows  for  trend  analysis  to  see  whether  rates  are  improving  or 
degrading  under  Better  Buying  Power  and  during  sequestration.  That  paper  found  that  while 
recent  reports  from  the  Government  Accountability  Office  found  reduced  competition, 
effective  competition  rates  have  held  steady  in  aggregate  despite  the  pressures  of 
sequestration.  The  paper  also  found  that  while  rates  of  effective  competition  for  products, 
services,  and  R&D  have  been  remarkably  consistent  over  time  for  the  DoD  as  a  whole, 
those  rates  differ  significantly  between  major  DoD  components.  In  particular,  there  are 
notable  differences  between  the  major  DoD  components  in  rates  of  effective  competition  for 
similar  categories  of  products,  services,  and  R&D,  which  is  an  issue  that  the  study  team 
intends  to  delve  into  further  in  the  next  stages  of  this  study. 

The  primary  focus  of  this  study  is  one  level  deeper:  Major  Contracting  Commands 
and  U.S.  states.  By  aggregating  the  data  at  this  level,  the  study  team  can  look  for  signs  of 
weakness  in  the  industrial  base  on  a  geographical  and  a  functional  basis.  The  utility  of  this 
study  is  to  go  beyond  obvious  findings,  such  as  low  levels  of  competition  for  complex 
defense  specific  systems,  to  instead  look  at  which  states  and  Major  Commands  are  over-  or 
underperforming  expectations.  To  do  that,  it  was  first  necessary  for  CSIS  to  set  a  baseline  of 
what  to  expect. 


1  CSIS  does  not  take  specific  policy  positions;  accordingly,  all  views  expressed  in  this  presentation 
should  be  understood  to  be  solely  those  of  the  author(s). 

2  Available  at  http://csis.orq/publication/qualitv-competition-defense-contracts-under-better-buyinq- 
power 
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To  set  that  baseline,  the  first  phase  of  this  study  analyzed,  in  isolation,  the 
relationship  between  a  variety  of  contract  characteristics  and  the  level  of  competition 
through  the  number  of  offers.  The  evaluated  characteristics  include  the  following: 

•  Type  of  Good  Procured:  Products,  Services,  and  Research  &  Development 
(R&D) 

•  Platform  Portfolios:  Land,  Air,  Vessel,  Missiles  and  Space,  Electronics  & 
Communications,  and  Weapons  and  Ammunition 

•  Contracting  Methods:  Contract  Pricing,  Contract  Size,  and  Contract  Vehicle 

Due  to  the  large  number  of  commands  and  U.S.  states,  the  results  from  that  first 
phase  are  available  at  the  Competition  repository  of  the  CSIS  defense  GitHub  account.3 
This  collection  includes  scatter  plots  for  each  study  variable  as  well  as  the  annual 
competition  rates  for  each  major  command  and  U.S.  state. 

After  examining  the  influence  of  contract  characteristics  in  isolation,  CSIS  proceeded 
to  study  their  interaction  on  two  separate  but  mutually  reinforcing  tracks.  The  first  track  was 
to  develop  a  regression  model  which  used  the  same  units  of  analyses  as  the  first  phase: 
major  commands  and  states.  For  each  geographical  and  organizational  unit,  the  study  team 
calculated  the  percentage  of  obligations  that  aligned  with  each  contract  characteristic  using 
a  denominator  of  constant  obligations  from  fiscal  year  (FY)2000  to  FY2013.  As  is  discussed 
below,  not  every  characteristic  could  be  included,  because  of  correlation.  Products  and 
services  inversely  correlate  with  one  another  while  Missiles  and  Space  correlates  positively 
with  R&D,  but  the  team  chose  those  contract  traits  which  had  the  highest  explanatory  value 
and  a  clear  causal  mechanism. 

On  a  parallel  track,  the  team  took  advantage  of  a  Bayesian  model  developed  for  a 
separate  research  project  on  fixed  price  contracting.  That  model  uses  individual  contracts, 
rather  than  major  commands  or  U.S.  states  as  the  unit  of  analysis  and  allows  for  use  of  the 
entire  dataset  of  publicly  reported  DoD  contracts  completed  between  FY2007  and  FY2013. 
This  larger  sample  approach  can  yield  more  statistical  power  and  is  well  suited  to  examining 
the  causal  connections  between  variables  in  addition  to  looking  at  how  those  connections 
influence  the  outcome. 

In  addition  to  studying  and  putting  in  place  enhanced  statistical  methods,  the  CSIS 
team  has  been  working  to  ensure  that  our  work  will  be  fully  reproducible  by  future 
researchers  who  seek  to  cross-validate  and  build  upon  our  results.  This  goes  beyond  the 
aforementioned  publishing  of  the  competition  data  for  each  U.S.  state  and  major  command 
and  work  that  was  completed  in  the  fall  of  2014  comparing  the  level  of  effective  competition 
for  each  individual  variable.  The  study  team  has  also  published  the  lookup  tables  used  to 
classify  the  Federal  Procurement  Data  System  (FPDS)  competition  data,  the  regression 
model  building  process,  and  the  Bayesian  model  and  analysis. 

This  paper  outlines  our  efforts  in  four  parts: 

•  Modeling  DoD  Effective  Competition  Rates  Using  Regression,  wherein 
we  detail  our  model  and  its  variables; 


3  Available  at  http://qithub.com/CSiSdefense/competition 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-319- 


•  Examination  of  Outlier  Major  Commands  and  U.S.  states,  which 
demonstrates  the  capabilities  of  the  CSIS  model  by  comparing  the  actual 
effective  rate  of  competition  with  the  predicted  rate  of  competition  on  a  select 
group  of  major  commands  and  states  of  interest; 

•  Tracing  the  Influence  of  Study  Variables  at  the  Contract  Level,  which 
takes  a  closer  look  at  the  independent  variables  included  in  the  regression 
model  and  analyzes  how  they  influence  competition  with  assistance  from  a 
Bayesian  network  that  uses  contracts  as  a  unit  of  analysis;  and 

•  Initial  Results  and  Planned  Next  Steps. 


Modeling  DoD  Effective  Competition  Rates  Using  Regression 

In  order  to  evaluate  trends  in  effective  competition  in  DoD  contracting,  CSIS  has 
divided  up  DoD  contracting  activity  in  two  ways: 


•  Place  of  Performance:  This  breakdown  examines  in  which  state  a  contract  is 
to  be  performed.  This  is  done  by  using  each  state  as  an  observation  and 
aggregating  the  data  from  each  state  for  each  variable  over  the  time  period 
from  2000  to  2013.  This  can  provide  indications  of  the  vibrancy  of  the 
industrial  base  available  to  perform  contracts  activity  that  takes  place  within  a 
particular  state.4 

•  Major  Contracting  Command  (MCC):  Going  a  level  below  the  usual  analytic 
level  of  “component,”  this  breakdown  allows  for  analysis  of  how  successful 
different  major  contracting  commands  have  been  in  promoting  effective 
competition  relative  to  the  goods  or  services  for  which  they  are  contracting 
and  the  types  of  contracts  for  which  they  are  responsible.  This  is  done  by 
using  each  MCC  as  an  observation  and  aggregating  the  data  from  each  MCC 
for  each  variable  over  the  time  period  from  2000  to  2013. 


For  these  two  data  groups,  CSIS  has  built  a  linear  regression  model  in  order  to 
generate  an  “estimated”  rate  of  effective  competition  for  each  state  and  MCC,  along  with  a 
confidence  interval.  This  model  uses  data  from  FY2000  to  2013  to  generate  a  prediction  of 
2014  effective  competition  rates,  based  on  a  number  of  variables  (described  in  the 
introduction,  and  explained  in  detail  in  a  later  section),  and  what  types  of  contracts  are  used. 
Each  of  these  variables  has  been  tested  and  validated  as  having  a  statistically  significant 
correlation  to  either  increasing  or  decreasing  rates  of  effective  competition. 

The  “estimated”  rate  of  effective  competition  is  then  compared  to  the  actual  rate  of 
effective  competition  for  FY2014.  At  this  point,  this  predictive  model  is  not  intended  to 
evaluate  performance  in  promoting  effective  competition — the  regression  model  is  only 
capable  of  explaining  about  half  of  the  variance  in  effective  competition  rates,  and  the 
confidence  intervals  for  estimates  were  sufficiently  wide  as  to  include  the  vast  majority  of 
actual  values  for  both  models.  CSIS  is  currently  refining  the  model  and  considering 


4  Because  the  issues  involved  with  competition  for  contracts  performed  overseas  and/or  in 
contingency  environments  are  notably  different  than  those  for  most  domestically-performed  contracts, 
contracts  whose  place  of  performance  is  outside  the  United  States  are  excluded  from  the  “place  of 
performance”  analysis.  CSIS  will  consider  whether  and  how  foreign  countries  should  be  included  for 
future  iterations. 
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increasing  the  sample  size  by  separating  out  the  MCC  and  state  entries  by  year.  For  the 
purposes  of  this  analysis,  results  from  the  regression  model  are  used  to  identify 
states/MCCs  where  the  difference  between  “estimated”  and  actual  effective  competition 
rates  is  notable  enough  to  warrant  deeper  investigation. 

Because  there  is  some  difference  in  the  variables  used  to  evaluate  states  versus 
those  used  for  MCCs,  the  following  discussion  is  divided  into  two  sections,  highlighting  a 
selection  of  states/MCCs  that  the  model  suggests  are  worthy  of  further  study. 

Major  Command  Regression 

The  following  variables  have  been  evaluated  as  being  part  of  a  statistically  significant 
causal  model  for  competition  in  MCCs;  however,  significance  varies  across  variables. 


lm(formula  =  pEff ecti vecomp  ~  pService:pProduct  +  pAir  +  pMnS  + 

pwnA  +  pEnc  +  piov  +  pservice  +  pservice :plDV  +  pProduct:pService:plDV, 
data  =  Mcclist; 

Residuals: 

Min  IQ  Median  3Q  Max 

-0.36330  -0.08168  0.02227  0.07601  0. 27716 


coefficients: 

(intercept) 

pAir 

pMns 

pwnA 

pEnc 

pIDV 

pservice 

pser vi ce : pproduct 

pservice :plDV 

pser vi ce : pproduct : plDV 

signif.  codes:  0  **** 


Estimate  std 
0.7161 
-0.4776 
-0.1763 
-0. 5896 
-0. 3050 
0.1038 
0.2678 
-2.8542 
-0.4570 
3. 5275 

0.001  '**’  0.01  **’ 


6.962  5. 19e-09  *** 
-3.276  0.001861  ** 
-1.217  0.228995 
-0.749  0.457136 
-2  .  64  6  0.010684  • 
0.750  0.456538 
1.614  0.112547 
-4.205  0.000101  *** 
-1.970  0.054064  . 
3.447  0.001119  ** 

0.05  '  0.1  *  ’  1 


Error  t  value  Pr(>|t|) 
0.1029 
0.1458 
0.1449 
0.7871 
0.1153 
0.1384 
0.1659 
0.6788 
0.2320 
1.0235 


Residual  standard  error:  0.1394  on  53  degrees  of  freedom 
Multiple  R-squared:  0.5987,  Adjusted  R-squared:  0.5305 
F-statistic:  8.784  on  9  and  53  DF,  p-value:  6.159e-08 


Variables  that  correlate  with  a  lower  rate  of  effective  competition: 

•  Greater  percentage  of  contract  obligations  related  to  Aircraft  &  Drone 
programs  (pAir);  Missiles  and  Space  programs  (pMnS);  Weapons  and 
Ammunition  programs  (pWnA);  and  Electronics  &  Communications  programs 
(pEnC). 

•  Greater  mix  of  contract  obligations  for  products  and  services 
(pService:pProduct).  As  the  contracting  portfolio  becomes  more  mixed,  rates 
of  effective  competition  decrease. 

•  Percentage  of  contract  obligations  for  services  and  awarded  under  IDV 
contract  types  (pService:plDV).  Competition  decreases  when  both  increase. 

Variables  that  correlate  with  a  higher  rate  of  effective  competition: 

•  Greater  percentage  of  contract  obligations  awarded  under  Indefinite  Delivery 
Vehicle  (IDV)  contract  types  (pIDV). 

•  Greater  percentage  of  contract  obligations  awarded  for  services  (pService). 
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•  Greater  mix  of  contracting  obligations  for  products  and  services,  along  with 
greater  percentage  of  contracts  awarded  under  IDV  contract  types 
(pService:pProduct:plDV). 

U.S.  State  Regression 

The  following  variables  have  been  evaluated  as  being  part  of  a  statistically  significant 
causal  model  for  competition  in  U.S.  states;  however,  significance  varies  across  variables. 

lm(formula  =  pEff ecti vecomp  ~  pAir  +  pMns  +  pEnC  +  plDV  +  pvessel  + 
pproduct  +  pEnc : pproduct ,  data  =  ShortstateList) 

Residuals : 

Min  IQ  Median  3Q  Max 

-0.158476  -0.069727  -0.000495  0.058374  0.284960 


coefficients: 

Estimate  std.  Error  t  value  Pr(>|t|) 


(intercept) 

0.78557 

0.08173 

9.611  1. 77e-12 

*** 

pAir 

-0.43742 

0.11976 

-3.652  0.000675 

*** 

pMns 

-0.05380 

0.  25996 

-0.207  0.836991 

pEnc 

-1.40027 

0. 39122 

-3. 579  0.000840 

★  *  * 

plDV 

0. 30292 

0.14010 

2.162  0.035957 

* 

pvessel 

0.06109 

0.15245 

0.401  0.690506 

pproduct 

-0.60550 

0.11802 

-5.131  5. 95e-06 

*** 

pEnc: pproduct 

1.80070 

0.67749 

2.658  0.010846 

* 

signif.  codes 

:  0  •***’ 

0.001 

0.01  *•*  0.05 

’  0.1  * 

Residual  standard  error 

:  0.09905  on  45  degrees  of  freedom 

Multiple  R-squared:  0. 

7023,  Adjusted  R-squared: 

0.656 

F-statistic: 

15.17  on  7 

and  45  DF, 

p-value:  5.431e 

-10 

Variables  that  correlate  with  a  lower  rate  of  effective  competition: 

•  Greater  percentage  of  contract  obligations  awarded  for  products  (pProduct) 

•  Greater  percentage  of  contract  obligations  related  to  Aircraft  &  Drone 
programs  (pAir);  Missiles  &  Space  programs  (pMnS);  Ships  programs 
(pVessel);  and  Electronics  &  Communications  (E&C)  programs  (pEnC)5 

Variables  that  correlate  with  a  greater  rate  of  effective  competition: 

•  Greater  percentage  of  contract  obligations  awarded  under  Indefinite  Delivery 
Vehicle  (IDV)  contract  types 

•  Greater  percentage  of  contract  obligations  awarded  for  both  products  and 
Electronics  &  Communications  (pProducts:pEnC;  competition  increases  when 
both  increase) 


5  These  groupings  include  all  product,  service,  and  R&D  contracts  associated  with  a  particular  class 
of  platforms  or  programs.  Where  relevant,  this  analysis  also  references  specific  categories  of 
products  or  services,  but  those  categories  are  specifically  noted  to  be  only  products  or  services  when 
mentioned. 
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Examination  of  Outlier  Major  Commands  and  U.S.  States 

After  having  created  the  predictive  regression  model  and  analyzed  the  causal  roots 
of  the  variables  that  drive  the  model,  the  study  team  put  these  tools  into  practice  to  study 
FY2014  contracting.  By  applying  the  regression  derived  from  FY2000  to  FY2013  data  to  the 
FY2014  inputs,  the  study  team  estimated  effective  competition  rates  for  each  of  the  states 
and  major  commands  in  the  sample  for  FY2014,  and  then  compared  the  predicted  value  to 
the  actual  effective  competition  rate  for  each  state  or  major  command  in  FY2014.  The  intent 
of  this  process  was  to  identify  the  organizational  and  geographical  units  that  exemplify  or 
defy  DoD-wide  patterns. 

At  this  stage  of  the  research  effort,  the  limits  of  the  predictive  strength  of  the  model 
means  that  the  following  analysis  is  not  intended  to  “grade”  MCCs  and  states  on  their 
competition  performance.  Rather,  the  following  sections  are  intended  as  a  proof  of  concept 
for  the  study  methodology,  demonstrating  the  ability  of  the  CSIS  model  to  identify  “outliers” 
among  the  MCCs  and  states.  The  drivers  of  the  differences  between  actual  and  “estimated” 
effective  competition  rates  for  the  identified  states  and  MCCs  will  be  used  to  improve  future 
iterations  of  the  CSIS  predictive  model. 

Major  Contracting  Commands6 

The  following  section  describes  five  MCCs,  four  with  significant  levels  of  contract 
obligations  that  show  notable  differences  between  “estimated”  effective  competition  rates 
and  actual  rates  in  2014  and  one  MCC  where  “estimated”  and  actual  rates  are  nearly 
identical. 

Defense  Logistics  Agency  (DLA)  Energy 

•  Total  2014  Contract  Obligations:  $8.4  billion 

•  “Estimated”  2014  Effective  Competition  Rate:  83% 

•  Actual  2014  Effective  Competition  Rate:  94% 

DLA  Energy  exceeded  its  “estimated”  competition  rate  in  2014  in  part  due  to  the 
unique  mix  of  goods  and  services  it  contracts  for,  as  well  as  higher  than  average  effective 
competition  rates  for  significant  parts  of  its  contracting  portfolio. 

As  would  be  expected,  the  vast  majority  of  contract  obligations  by  DLA  Energy  are 
for  fuels.  Approximately  96%  of  fuels  contract  obligations  by  DLA  Energy  are  awarded  after 
effective  competition  in  2014,  right  in  line  with  the  overall  DoD  rate  of  95%.  DLA  Energy  also 
handles  significant  contract  obligations  for  facilities-related  services  and  construction 
(FRS&C;  primarily  for  lease  and  operation  of  fuel  supply  facilities,  as  well  as  utilities),  and 
88%  of  those  contract  obligations  were  awarded  after  effective  competition.  By  comparison, 
the  rate  of  effective  competition  for  overall  DoD  FRS&C  contract  obligations  is  70%.  Over 
92%  of  DLA  Energy  contract  obligations  are  awarded  under  IDV  contract  types,  which  are 
associated  with  higher  rates  of  effective  competition.  Most  of  the  other  correlative  variables 
do  not  apply  to  DLA  Energy,  which  may  explain  the  gap  between  the  “estimated”  and  actual 
effective  competition  rates. 


6  Because  the  95%  confidence  intervals  for  the  MCC  model  are  so  wide,  no  MCCs  fell  outside  of  their 
intervals.  CSIS  sees  this  not  as  a  sign  of  the  predictive  power  of  the  model,  but  rather  a  limitation  to 
be  addressed  in  future  iterations. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-323- 


DLA  Aviation 


•  Total  2014  Contract  Obligations:  $4.3  billion 

•  “Estimated”  2014  Effective  Competition  Rate:  42% 

•  Actual  2014  Effective  Competition  Rate:  20% 

DLA  Aviation  significantly  underperformed  its  “estimated”  effective  competition  rate  in 
2014,  due  to  lower  than  average  rates  of  effective  competition  for  Electronics  & 
Communications,  Facilities  &  Construction,  and  products  categorized  as  “Engines  &  Power 
Plants.” 


Unsurprisingly,  the  majority  of  DLA  Aviation  contract  obligations  are  for  Aircraft  & 
Drones,  with  a  significant  share  also  going  for  Electronics  &  Communications;  both  correlate 
to  lower  rates  of  effective  competition.  The  DLA  Aviation  rate  of  effective  competition  for 
Aircraft  &  Drones  contract  obligations  was  low  in  2014  (17%),  but  that  is  only  slightly  lower 
than  the  rate  for  Aircraft  &  Drones  contracts  in  the  DoD  overall  (21%).  (Within  Aircraft  & 
Drones,  only  12%  of  DLA  Aviation  contract  obligations  for  products  categorized  as  “Engines 
&  Power  Plants”  were  awarded  after  effective  competition,  compared  to  37%  for  the  DoD 
overall.)  By  contrast,  the  rate  of  effective  competition  for  Electronics  &  Communications 
within  DLA  Aviation  (22%)  is  significantly  lower  than  the  rate  for  the  DoD  overall  (35%). 

Interestingly,  while  73%  of  overall  DoD  contract  obligations  for  products  and  services 
categorized  as  “Facilities  &  Construction”  (F&C)  are  awarded  after  effective  competition,  that 
rate  is  only  36%  for  F&C  contract  obligations  within  DLA  Aviation. 

Army  Materiel  Command  (AMC) 

•  Total  2014  Contract  Obligations:  $43.6  billion 

•  “Estimated”  2014  Effective  Competition  Rate:  34% 

•  Actual  2014  Effective  Competition  Rate:  46% 

The  main  driver  of  AMC’s  higher-than-“estimated”  rate  of  effective  competition  in 
2014  is  the  unusually  high  rate  of  effective  competition  for  one  category  of  its  services 
contracts  portfolio. 

Based  solely  on  the  correlative  variables  for  AMC,  AMC’s  effective  competition  rates 
being  higher  than  “estimated”  in  2014  seems  unusual:  a  nearly  even  mix  of  products  and 
services,  as  well  as  higher-than-average  shares  of  obligations  going  to  Aircraft  &  Drones 
and  Electronics  &  Communications,  correlate  with  lower  rates  of  competition.  The  main 
driver  of  the  higher-than-“estimated”  rate  of  effective  competition  for  AMC  seems  to  be  in  the 
rate  of  effective  competition  for  professional,  administrative,  and  management  support 
services  (PAMS):  PAMS  account  for  nearly  a  quarter  of  AMC’s  contract  portfolio,  and  66% 
were  awarded  after  effective  competition,  compared  to  36%  for  the  DoD  overall. 

Naval  Supply  Systems  Command  (NAVSUP) 

•  Total  2014  Contract  Obligations:  $7.1  billion 

•  “Estimated”  2014  Effective  Competition  Rate:  52% 

•  Actual  2014  Effective  Competition  Rate:  33% 

The  main  drivers  of  the  lower-than-predicted  rate  of  effective  competition  are  twofold. 
First,  only  3%  of  NAVSUP  contract  obligations  for  Aircraft  &  Drones  in  2014  were  awarded 
after  effective  competition,  compared  to  21%  for  the  DoD  overall.  Secondly,  only  23%  of 
NAVSUP  contract  obligations  for  Electronics  &  Communications  were  awarded  after 
effective  competition,  compared  to  45%  for  the  DoD  overall. 
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The  relevant  correlative  variables  for  NAVSUP  are  a  mixed  bag:  a  relatively  even 
mix  of  products  and  services,  along  with  slightly  higher-than-average  shares  of 
obligations  going  to  Aircraft  &  Drones  and  Electronics  &  Communications,  correlate 
with  lower  rates  of  effective  competition.  Meanwhile,  higher-than-average  usage  of 
IDV  contract  types,  especially  alongside  the  aforementioned  even  mix  of  products 
and  services,  correlate  with  higher  rates  of  effective  competition. 

Air  Force  Materiel  Command  (AFMC) 

•  Total  2014  Contract  Obligations:  $36.7  billion 

•  “Estimated”  2014  Effective  Competition  Rate:  30% 

•  Actual  2014  Effective  Competition  Rate:  30% 

Unlike  the  previous  four  examples,  where  particular  categories  of  contracts  have 
significantly  higher/lower  rates  of  effective  competition  than  the  DoD  overall  in  one  direction, 
there  are  notable  differences  in  both  directions  for  AFMC. 

AFMC  has  a  high  percentage  of  contract  obligations  going  to  Aircraft  &  Drones 
(though,  at  38%,  not  as  high  as  one  might  assume),  which  correlates  with  lower  rates  of 
effective  competition.  AFMC  also  has  a  relatively  even  mix  of  products  and  services  in  its 
contracting  portfolio,  which  similarly  correlates  with  lower  competition  rates.  Only  26%  of 
AFMC  contract  obligations  for  PAMS  in  2014  were  awarded  after  effective  competition, 
compared  to  36%  overall.  By  contrast,  for  AFMC  equipment-related  services  (ERS),  33%  of 
contract  obligations  were  awarded  after  effective  competition,  compared  to  26%  overall. 

This  lack  of  unidirectional  deviations  from  “estimated”  rates  of  competition  is  likely  a 
significant  factor  enabling  the  predictive  model  to  accurately  estimate  2014  effective 
competition  rates. 

Place  of  Performance — States 

The  following  sections  describe  four  states  with  significant  levels  of  contract 
obligations  that  show  notable  differences  between  “estimated”  effective  competition  rates 
and  actual  rates  in  2014,  along  with  one  state  where  “estimated”  and  actual  rates  are  nearly 
identical. 

Minnesota 

•  Total  2014  Contract  Obligations:  $3.9  billion 

•  “Estimated”  2014  Effective  Competition  Rate:  65% 

•  Actual  2014  Effective  Competition  Rate:  89%  (Upper  limit  of  95%  confidence 
interval:  89%) 

DoD  contract  obligations  performed  in  Minnesota  significantly  over-performed  their 
“estimated”  effective  competition  rates,  due  to  higher  than  average  rates  of  effective 
competition  for  medical  services. 

Only  19%  of  contract  obligations  for  contracts  performed  in  Minnesota  were  for 
products  in  2014,  compared  to  45%  for  the  DoD  overall,  which  would  correlate  with  higher 
rates  of  effective  competition.  At  the  same  time,  only  11%  of  contract  obligations  were 
awarded  under  IDV  contract  types,  less  than  a  quarter  of  the  rate  for  the  overall  DoD,  which 
would  correlate  with  lower  competition.  Nearly  80%  of  obligations  for  contracts  performed  in 
Minnesota  were  for  medical  services,  specifically  “General  Health  Care  Services.” 
Interestingly,  nearly  all  of  those  medical  services  contract  obligations  were  awarded  after 
receiving  only  two  offers;  while  the  overall  rate  of  competition  is  exceptional,  this  is  not 
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necessarily  indicative  of  a  highly  competitive  marketplace.  For  overall  DoD  medical  services 
contracting  in  2014,  approximately  75%  of  contract  obligations  were  awarded  after  effective 
competition,  with  the  vast  majority  of  those  receiving  five  or  more  offers. 

Mississippi 

•  Total  2014  Contract  Obligations:  $2.3  billion 

•  “Estimated”  2014  Effective  Competition  Rate:  44% 

•  Actual  2014  Effective  Competition  Rate:  71%  (Upper  limit  of  95%  confidence 
interval:  67%) 

DoD  contract  obligations  performed  in  Mississippi  significantly  outperformed  their 
“estimated”  effective  competition  rates,  due  to  higher  than  average  rates  of  effective 
competition  for  contracts  related  to  Aircraft  and  Ships  programs. 

A  disproportionately  large  share  (42%)  of  contract  obligations  performed  in 
Mississippi  in  2014  were  related  to  Ships  platforms,  primarily  related  to  construction  of  DDG- 
51  destroyers  at  Ingalls  Shipbuilding  in  Pascagoula,  MS,  owned  by  Huntington  Ingalls 
Industries.  Nearly  three-quarters  of  Ships  obligations  were  awarded  after  effective 
competition,  almost  entirely  with  only  two  offers,  which  is  largely  a  function  of  the  limited 
industrial  base  for  large  combat  ships.  Approximately  30%  of  contract  obligations  performed 
in  Mississippi  were  related  to  Aircraft  programs,  which  generally  have  low  levels  of  effective 
competition,  due  to  the  limited  industrial  base  for  aircraft  platforms,  the  long-term  nature  of 
aircraft  programs,  and  the  tendency  for  maintenance  contracts  to  be  performed  by  the 
development/production  vendor.  But  the  contract  obligations  in  Mississippi,  mainly  for 
“maintenance-repair  of  aircraft,”7  were  highly  competitive,  with  nearly  three-quarters 
awarded  after  effective  competition,  split  relatively  evenly  between  two  offers  and  three  or 
more  offers. 

Nevada 

•  Total  2014  Contract  Obligations:  $1.3  billion 

•  “Estimated”  2014  Effective  Competition  Rate:  69% 

•  Actual  2014  Effective  Competition  Rate:  41%  (Lower  limit  of  95%  confidence 
interval:  47%>) 

DoD  contract  obligations  performed  in  Nevada  notably  underperformed  their 
“estimated”  effective  competition  rates,  primarily  due  to  a  lower  than  average  rate  of 
effective  competition  for  one  category  of  services. 

Contract  obligations  for  contracts  performed  in  Nevada  are  overwhelmingly  IDV-type 
contracts  (80%)  in  2014,  which  correlate  with  higher  levels  of  effective  competition,  and  only 
24%  of  contracts  are  for  products,  which  is  well  below  the  levels  for  the  overall  DoD. 
Nonetheless,  the  rate  of  effective  competition  for  contracts  performed  in  Nevada  in  2014  is 
well  below  the  “estimated”  rate  produced  by  the  regression  model  and  significantly  below 
the  lower  limit  of  the  95%  confidence  interval.  The  main  source  of  this  low  rate  of 
competition  seems  to  be  an  18%  effective  competition  rate  for  PAMS  performed  in  Nevada, 


7  These  obligations  are  likely  related  to  Columbus  Air  Force  Base,  a  major  Air  Force  pilot  training 
installation. 
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around  half  the  rate  seen  for  PAMS  in  the  DoD  overall;  PAMS  account  for  approximately 
40%  of  contract  obligations  performed  in  Nevada.  Most  of  the  obligations  under  PAMS  are 
for  “logistics  support  services.” 

Washington 

•  Total  2014  Contract  Obligations:  $7.5  billion 

•  “Estimated”  2014  Effective  Competition  Rate:  13% 

•  Actual  2014  Effective  Competition  Rate:  45%  (Upper  limit  of  95%  confidence 
interval:  37%) 

DoD  contract  obligations  performed  in  Washington  state  significantly  over-performed 
their  “estimated”  effective  competition  rates  in  2014,  primarily  due  to  higher  than  average 
rates  of  effective  competition  for  contracts  related  to  Aircraft  programs  and  contracts  for 
facilities-related  services  &  construction. 

As  would  be  expected  from  the  home  state  of  Boeing,  a  high  share  (66%)  of 
obligations  for  contracts  performed  in  Washington  go  to  Aircraft  &  Drones  programs,  which 
are  associated  with  lower  rates  of  effective  competition.  A  below-average  share  (26%)  of 
Washington  contract  obligations  are  IDV-type  contracts,  which  would  also  correlate  with  less 
competition.  The  source  of  the  higher-than-“estimated“  rate  of  competition  for  contracts 
performed  in  Washington  appears  to  be  a  higher-than-average  rate  of  competition  for 
Aircraft-related  products:  While  only  10%  of  contract  obligations  for  the  DoD  overall  were 
awarded  after  effective  competition  in  2014,  35%  of  those  contracts  performed  in 
Washington  were  awarded  after  effective  competition.8 

FRS&C  also  makes  up  a  significant  share  of  contracts  performed  in  Washington,  and 
the  data  shows  a  highly  competitive  market:  88%  of  Washington  FRS&C  contract 
obligations  were  awarded  after  effective  competition  in  2014,  compared  to  70%  for  the  DoD 
overall.  And  nearly  90%  of  the  effectively  competed  FRS&C  contracts  performed  in 
Washington  received  three  or  more  offers. 

Virginia 

•  Total  2014  Contract  Obligations:  $33.6  billion 

•  “Estimated”  2014  Effective  Competition  Rate:  54% 

•  Actual  2014  Effective  Competition  Rate:  55% 

Despite  a  higher  than  average  rate  of  effective  competition  for  a  category  of  services 
that  makes  up  a  significant  share  of  contract  obligations  performed  in  Virginia,  the  predictive 
model  was  able  to  almost  exactly  predict  Virginia’s  effective  competition  rate  for  2014. 

Only  32%  of  obligations  for  contracts  performed  in  Virginia  were  for  products, 
compared  to  41%  overall,  which  would  tend  to  correlate  with  higher  rates  of  effective 
competition.  Similarly,  67%  of  Virginia’s  contract  obligations  were  awarded  under  IDV 
contract  types,  notably  higher  than  the  rate  for  the  DoD  overall.  As  might  be  expected  given 


8  Due  to  poor  data  labeling,  the  competed  portion  of  Washington’s  Aircraft  contract  obligations  are 
labeled  as  being  associated  with  the  Shillelagh  Missile,  a  1970s  Army  anti-tank  missile  program. 
CSIS  is  engaging  with  experts  to  try  to  determine  what  program  these  contract  obligations  are 
actually  associated  with. 
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the  volume  of  available  vendors,  the  market  for  PAMS  in  Virginia  is  significantly  more 
competitive  than  it  is  nationwide:  61%  of  PAMS  contract  obligations  performed  in  Virginia 
were  awarded  after  effective  competition,  compared  to  36%  nationwide. 

It  is  also  notable  that,  for  R&D  contract  obligations  performed  in  Virginia,  36% 
awarded  after  competitions  received  only  a  single  offer — over  twice  the  rate  for  R&D 
nationwide.  As  a  result,  the  rate  of  effective  competition  for  R&D  contracts  performed  in 
Virginia  was  only  32%  in  2014,  compared  to  46%  nationwide.  Given  the  heavy  concentration 
of  major  R&D  vendors  in  Virginia,  this  high  rate  of  single-offer  competition  is  likely  masking 
contracts  that  would  be  more  properly  classified  as  noncompetitive. 

Tracing  the  Influence  of  Study  Variables  Using  a  Bayesian  Model 

For  predictive  analysis  via  regression  to  be  most  effective,  it  is  important  to 
understand  the  causal  mechanisms’  underlying  analysis.  For  this  reason,  the  study  team 
examined  each  of  the  variables  used  in  the  regression  at  the  contract  level.  This  is  done 
using  a  smaller  subset  of  the  FPDS  database,  six  million  publicly  reported  DoD  contracts 
that  were  completed  between  FY2007  and  FY2013.  While  this  population  of  contracts 
includes  fewer  years,  because  the  unit  of  analysis  is  contracts  rather  than  major  commands 
or  states,  the  number  of  observations  is  dramatically  higher.  For  similar  reasons,  the 
effective  competition  rates  are  different  when  calculating  by  market  share  of  obligations  than 
when  calculating  by  number  of  contracts. 

The  process  of  gathering,  cleaning,  and  organizing  the  data  for  the  Bayesian  network 
is  described  in  greater  detail  in  the  CSIS  report  on  fixed-price  contracts  that  is  also  being 
presented  at  the  2015  Acquisition  Research  Symposium.  In  summary,  the  study  team 
categorized  and  then  collated  FPDS  data  on  a  contract  basis,  a  process  which  required 
significant  error  correction.  The  end  product  of  this  process  is  1 1  evidence  nodes,  each 
having  between  two  and  eight  distinct  states.  The  study  team  then  created  the  Bayesian 
network  shown  in  Figure  1.  In  that  model,  an  evidence  node  is  a  circle,  and  each  arc 
indicates  the  direction  that  influence  flows.  The  1 1  nodes  are  described  in  the  following 
section.9 


9  See  the  Fixed  Price  repository  under  the  CSISdefense  GitHub  account  for  the  processing  code  used 
to  create  the  model  as  well  as  the  data  itself.  The  nature  of  the  Bayesian  model  is  described  in 
greater  detail  in  the  fixed-price  report. 
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Figure  1.  Competition  Bayesian  Network 
Contract  Fundamentals 

These  nodes  of  evidence  include  both  the  types  of  good  procured  and  the  platform 
portfolio.  The  methodological  description  is  adapted  from  the  fixed-price  paper,  which 
provides  greater  detail. 

Who  (Customer:  Army/Navy/Air  Force/Other  DoD):  Determined  by  the  contracting 
office  rather  than  the  funding  office. 

What  (Platform:  Air/Land/Vessel/Electronics  &  Communications/Missiles  and 
Space/Weapons  and  Ammunition/Facility  Related  Services  &  Construction/Other): 

Determined  by  the  combination  of  the  claimant  program  code  for  the  platform  when 
available  and  otherwise  via  the  product  or  service  code. 

PSR  (Product/Service/R&D):  Determined  by  the  product  or  service  codes,  with 
R&D  management  and  support  being  treated  as  a  service. 

Inti  (International:  Just  U.S./Some  International):  Based  on  the  place  of 
performance.  Those  contracts  with  any  transactions  in  foreign  countries  are  treated  as 
having  transactions  with  an  international  place  of  performance. 

Link  (#  of  linked  contracts:  none/1-749/750+):  This  calculated  column  is  the  study 
team’s  first  effort  to  account  for  the  possibility  of  problems  cascading  from  other  related 
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contracts.  This  is  calculated  by  looking  at  contracts  with  the  same  equipment  code  or  for 
contracts  with  the  same  platform  portfolio  (excluding  the  FRS&C  and  other  platforms)  and 
contracting  office,  with  provisions  to  avoid  double  counting  or  counting  contracts  from 
different  MDAPs.  See  the  Fixed-price  repository  of  the  CSISdefense  GitHub  account  for 
processing  code  used  for  this  calculation.10 

Contract  Approach 

These  evidence  nodes  include  the  contracting  methods  discussed  in  earlier  sections 
as  well  as  scope-related  parameters  such  as  the  contract  ceiling  and  the  initial  duration.  The 
regression  did  consider  variables  such  as  average  contract  size.  However,  the  predictive 
value  of  contract  size  is  notably  more  effective  at  the  contract  level  than  when  aggregated  at 
the  state  or  major  command  level. 

Comp  (Competition:  Comp./No  Comp.):  Is  determined  using  the  standard  CSIS 
methodology  with  the  critical  exception  that  the  numbers  of  offers  received  is  treated  as  a 
separate  piece  of  evidence. 

Ceil  (Ceiling:  $15,000/$100,000/$1,000,000/$30,000,000):  Refers  to  the  initial 
ceiling  on  total  potential  contract  obligations.  Is  set  by  the  initial  base  and  all  options  value 
for  the  contract.  This  value  was  chosen  rather  than  the  initial  base  and  exercised  options 
value  because  exercising  options  happens  regularly  during  the  course  of  an  on-time  and  on- 
budget  contract. 

Dur  (Duration:  One  Day  to  Two  Months/Seven  Months  to  a  Year/More  than  a 
Year):  Refers  to  the  duration  and  is  calculated  using  the  number  of  days  between  the  initial 
effective  date  and  the  current  completion  date  for  the  contract.  The  ultimate  completion  date 
is  also  available  but  was  regularly  unlabeled. 

FxCb  (Fixed-price  or  Cost-based):  Fixed-price  includes  all  forms  of  fixed-price 
contracting  except  fixed-price  level  of  effort.  That  comparatively  rare  form  has  been 
described  in  meetings  with  DoD  officials  as  exhibiting  more  properties  of  cost-based 
contracts.  Cost-based  includes  all  forms  of  cost-plus  contracts  as  well  as  time  and  materials 
and  labor  hours  contracts. 

Indefinite  Delivery  Vehicle  (IDV):  Indicates  whether  or  not  a  contract  is  one  of  the 
many  forms  of  Indefinite  Delivery  Vehicles  (IDVs).  This  is  a  contracting  approach  in  which  a 
single-root  contract  is  used  as  a  basis  for  multiple  other  contracts. 

Contract  Outcomes 

Offr  (Number  of  Offers  Received:  1,  2,  3-4,  5+):  This  final  evidence  node  is  the 
one  of  most  interest  to  this  study.  Cases  with  no  competition  are  categorized  as  only  one 
offer  and  further  details  of  the  calculation  of  this  field  are  covered  in  the  Fixed-price  paper. 

Figure  2  shows  the  distribution  of  contracts  by  number  of  offers,  faceted  by  the  initial 
contract  ceiling  and  use  of  an  indefinite  delivery  vehicle  (IDV).  While  all  five  categories  show 
a  similar  broad  curve,  larger  and  smaller  ceiled  contracts  do  have  clearly  different  patterns. 
There  is  a  large  spike  at  exactly  five  offers  for  those  contracts  with  ceilings  below  $1  million. 
That  spike  is  even  more  prominent  for  those  with  ceilings  below  $100  thousand.  As  is  shown 


10  Available  at  https://qithub.com/CSISdefense/Fixed-price 
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by  comparing  the  left  and  right  columns  of  Figure  2  this  phenomenon  is  entirely  driven  by 
IDV  contracts  and  given  the  immediate  drop  off  from  five  to  six  contracts,  this  data  would  be 
consistent  with  a  widespread  coding  quirk  or  a  deliberate  if  not  necessarily  regulatory  target 
for  contracting  officers  dealing  with  smaller  IDVs  contracts. 
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Note.  Contracts  with  more  than  50  offers  are  excluded  from  the  graph.  The  data  in  this  figure  comes  from  FPDS, 
CSIS  analysis. 


Figure  2.  Number  of  Offers  by  Ceiling  and  Vehicle 

By  comparison,  single-offer  competition  is  prominent  in  all  categories  and  exceeds 
two  offers  for  contracts  with  ceilings  between  $100,000  and  $30  million.  As  is  shown  in 
Figure  3,  this  phenomenon  is  driven  first  and  foremost  by  definitive  contracts  and  purchase 
orders,  which  below  the  $30  million  dollar  ceiling  level  consistently  has  more  single-offer 
competition  than  any  other  number  of  offers.  On  the  IDV  side,  this  is  only  true  for  contracts 
with  ceilings  greater  than  or  equal  to  $1  million.  For  both  categories  the  median  number  of 
offers  is  lowest  for  contracts  with  ceilings  between  $100,000  and  $30  million. 
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Box  plot  of  number  of  offers  received  by  celling  and  IDV 
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a  a  a 

Initial  contract  celling 


Note.  Contracts  with  more  than  30  offers  are  excluded  from  the  graph.  The  mean  number  of  offers  for  IDV 
contracts  with  ceilings  of  $30  million  or  higher  is  33.4  and  not  depicted  on  this  graph.  The  data  in  this  figure 
comes  from  FPDS,  CSIS  analysis. 

Figure  3.  Box  Plot  of  Number  of  Offers  Received  by  Ceiling  and  IDV 

However,  above  $30  million,  single  offer  drops  off  as  a  proportion  and  upper 
quadrants  of  the  box  plot  reach  their  highest  values.  As  depicted  on  the  graph,  at  least  a 
quarter  of  these  large  contracts  receive  more  than  five  offers  regardless  of  vehicle.  This 
would  be  consistent  with  middle  tier  contracts  sometimes  attracting  fewer  offers  when  they 
are  unappealing  or  overly  fitted  for  a  single  vendor.  However,  once  the  contracts  get  above 
the  $30  million  threshold,  vendors  appear  to  be  generally  more  willing  to  bid  even  though 
the  potential  risks  are  greater  in  absolute  terms. 

As  the  above  summary  indicates,  the  number  of  offers  received  varies  considerably 
depending  on  the  characteristics  of  the  contract.  The  patterns  of  lower  competition  for 
contracts  between  $100,000  and  $1  million  suggests  that  there  is  a  middle  tier  of  contracts 
which  are  not  so  small  as  to  be  easily  pursued  or  so  large  as  to  be  actively  enticing  to  a 
large  number  of  competitors.  The  strangest  finding  is  the  prevalence  of  five  offer  competition 
for  IDVs,  which  merits  further  investigation. 
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Contract-Level  Analysis  of  How  the  Study  Variables  Influence  Effective  Competition 

Type  of  Good  Procured  (PSR) 

Services 

For  MCCs,  increasing  contract  obligations  for  services  correlate  with  higher  rates  of 
effective  competition.  The  correlative  strength  within  the  MCC  model  for  services  is  weak-to- 
moderate  (p  =  0.1 12547),  as  is  the  degree  of  impact11  (0.2678). 

Overall,  55%  of  contracts  for  services  in  2014  were  awarded  after  effective 
competition.  For  Aircraft  services,  however,  that  rate  was  only  36%.  By  contrast,  two  of  the 
other  control  variables  are  associated  with  significantly  higher  rates  of  effective  competition 
for  services:  66%  of  services  contracts  with  a  large  contract  ceiling  were  effective  competed, 
while  61  %  of  services  contracts  structured  as  I DVs  were  awarded  after  effective 
competition. 

Products 

The  share  of  contract  obligations  for  products  correlates  with  lower  rates  of  effective 
competition  for  states  but,  does  not  correlate  within  the  model  with  rates  of  effective 
competition  as  strongly  as  does  the  share  going  to  services  for  MCCs.  The  reverse  was  true 
for  states.  In  each  case  the  variable  with  higher  correlation  was  used.  The  correlative 
strength  within  the  states  model  for  products  is  strong  (p  =  5.95e-06),  and  its  degree  of 
impact  is  moderate  (-0.60550). 

Overall,  67%  of  DoD  contracts  for  products  were  awarded  after  effective  competition 
in  2014. 12  A  few  of  the  control  variables  had  notable  effects  on  the  rate  of  effective 
competition  for  products:  only  47%  of  Aircraft  products  contracts  were  awarded  after 
effective  competition;  for  products  contracts  with  a  contract  ceiling  over  $1  million,  58%  of 
were  awarded  after  effective  competition;  74%  of  products  contracts  that  were  IDVs  were 
awarded  after  effective  competition;  and  only  42%  of  products  contracts  of  long  duration 
were  awarded  after  effective  competition. 

Contracting  Methods  (IDV) 

IDVs 

Increasing  shares  of  contract  obligations  awarded  under  IDV  contract  types  correlate 
with  increasing  rates  of  effective  competition,  but  the  strength  of  those  correlations  is 
notably  different  between  states  and  MCCs.  For  states,  the  correlative  strength  is  strong  (p 
=  0.035957),  and  the  degree  of  impact  within  the  model  is  mild  (0.  30292).  For  MCCs,  the 
correlative  strength  is  significantly  weaker  (p  =  0.456538),  and  the  degree  of  impact  is  even 
lower  (0.1038). 

Overall,  71%  of  contracts  awarded  under  IDV  contract  types  in  2014  were  effectively 
competed,  but  association  with  the  control  variables  had  significant  effects  on  rates  of 


11  The  study  team  uses  “degree  of  impact”  to  describe  the  slope  of  the  line  resulting  from  the 
correlative  analysis  between  the  specified  variable  and  rates  of  effective  competition  within  the  model. 

12  Note  again  that  this  refers  to  the  share  of  contracts  awarded  after  effective  competition,  not  the 
share  of  contract  obligations.  By  contract  obligations,  the  rate  of  effective  competition  for  DoD 
products  is  consistently  well  below  the  rate  for  services.  By  contracts,  however,  the  sheer  volume  of 
smaller  contracts  for  commercial-type  products  skews  the  results. 
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effective  competition  for  IDVs.  IDVs  related  to  Aircraft  (38%),  I  DVs  with  large  contract 
ceilings  (66%),  IDVs  with  ceilings  over  $1  million  (60%),  and  IDVs  of  long  duration  (48%)  all 
showed  lower  rates  of  effective  competition  than  overall  IDVs. 

Platform  Portfolios  (What) 

Electronics  &  Communications 

For  both  states  and  MCCs,  increasing  contract  obligations  related  to  Electronics  & 
Communications  (E&C)  correlate  with  lower  rates  of  effective  competition,  but  the  strength 
of  that  correlation  differs  for  states  and  MCCs.  The  correlative  strength  within  the  model  of 
E&C  for  states  is  high  (p  =  0.000840),  and  the  degree  of  impact  is  the  highest  of  any 
variable  included  (-1.40027).  For  MCCs,  by  contrast,  the  correlative  strength  within  the 
model  of  E&C  is  relatively  weak  (p  =  0.13066),  and  the  degree  of  impact  is  similarly  lower 
(-0.123549).  This  difference  between  states  and  MCCs  may  have  two  causes.  First,  if  a 
wider  number  of  states  do  E&C  work,  that  would  make  the  model  less  dependent  on  any 
given  outlier.  Second,  for  states  an  interaction  variable  was  included.  Electronics  and 
communications  products  were  an  exception  to  the  negative  trend  and  correlated  with 
greater  levels  of  effective  competition  (p=0.010846),  with  a  high  degree  of  impact  (1.80070) 
perhaps  because  services  in  electronics  and  communications  are  much  more  technically 
involved. 

Overall,  55%  of  contracts  (not  contract  obligations)  for  E&C  were  awarded  after 
effective  competition  in  2014.  Looking  at  the  control  variables  to  see  which  had  a  significant 
effect  on  effective  competition  rates  for  E&C,  contract  ceiling  and  whether  the  contract  was 
an  IDV  had  virtually  no  effect.  But  contracts  for  E&C  of  long  duration  at  contract  start  had  an 
effective  competition  rate  of  only  46%,  likely  reflecting  contracts  with  greater  complexity  that 
fewer  companies  were  willing  or  able  to  bid  on. 

Aircraft 

The  share  of  contract  obligations  related  to  Aircraft  strongly  correlates  with  lower 
rates  of  effective  competition  for  both  states  and  MCCs.  For  MCCs,  the  correlative  strength 
is  high  (p  =  0.001861),  and  the  degree  of  impact  within  the  model  was  moderately  high  (- 
0.4776).  For  states,  the  correlative  strength  was  even  higher  (p  =  0.000675),  and  the  degree 
of  impact  within  the  model  was  slightly  smaller  (-0.  43742). 

In  2014,  46%  of  contracts  related  to  Aircraft  were  awarded  after  effective 
competition.  In  three  cases,  the  control  variables  were  associated  with  lower  rates  of 
effective  competition  for  Aircraft.  For  Aircraft  contracts  with  a  contract  ceiling  of  greater  than 
$1  million,  the  rate  of  effective  competition  was  only  36%.  For  Aircraft  contracts  structured 
as  IDVs,  only  38%  were  effectively  competed.  And  for  Aircraft  contracts  of  long  duration, 
only  34%  were  awarded  after  effective  competition. 

Missiles  and  Space 

The  share  of  contract  obligations  related  to  Missiles  and  Space  (M&S)  is  weakly-to- 
moderately  correlated  with  lower  rates  of  effective  competition  for  both  states  and  MCCs. 
The  correlative  strengths  were  superior  for  MCCs  (p  =  0.  836991  for  states;  p  =  0.228995  for 
MCCs)  and  in  the  same  manner  the  degree  of  impact  higher  for  MCCs  (-0.1763)  than  states 
(-0.  05380). 

Overall,  49%  of  contracts  related  to  M&S  in  2014  were  awarded  after  effective 
competition.  Looking  at  the  interaction  between  M&S  and  the  control  variables,  only  40%  of 
M&S  contracts  with  large  contract  ceilings  were  awarded  after  effective  competition.  By 
contrast,  54%  of  M&S  contracts  of  long  duration  were  awarded  after  effective  competition. 
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Ships 

The  share  of  contract  obligations  related  to  Ships  did  not  notably  correlate  with  rates 
of  effective  competition  for  MCCs  and  showed  a  weak-to-moderate  correlation  with  lower 
rates  of  effective  competition  for  states.  The  correlative  strength  of  Ships  for  ships 
(0.690506)  is  slightly  higher  than  for  M&S,  although  the  degree  of  impact  within  the  model 
(0.06109)  is  minimal.  Due  to  the  surprising  sign  of  the  correlation  and  the  weak  significance, 
this  variable  may  not  merit  inclusion  in  future  iterations  of  the  model. 

Overall,  46%  of  contracts  related  to  Ships  were  awarded  after  effective  competition 
in  2014.  Several  of  the  control  variables  are  associated  with  notably  different  rates  of 
effective  competition  for  Ships.  69%  of  Ships  contracts  with  large  ($30  million  or  higher) 
contract  ceilings  were  awarded  after  effective  competition.  Meanwhile,  53%  of  Ships 
contracts  with  a  contract  ceiling  over  $1  million  saw  effective  competition.  Similarly,  52%  of 
Ships  contracts  awarded  under  IDVs  were  awarded  after  effective  competition.  By  contrast, 
only  26%  of  Ships  contracts  of  long  duration  were  effectively  competed. 

Weapons  and  Ammunition 

The  share  of  contract  obligations  related  to  Weapons  and  Ammunition  (W&A) 
showed  weak  with  lower  shares  of  effective  competition  for  MCCs.  For  MCCs,  the 
correlative  strength  was  notably  weaker  (p  =  0.457136),  but  the  degree  of  impact  within  the 
model  was  the  highest  of  any  single  variable  (-0.5896). 

Overall,  57%  of  contracts  related  to  W&A  were  awarded  after  effective  competition  in 
2014.  The  only  control  variable  that  had  a  significant  impact  in  association  with  W&A  was 
long  duration — W&A  contracts  of  long  duration  were  effectively  competed  only  52%  of  the 
time. 

Initial  Results  and  Next  Steps 

Due  to  the  previously-mentioned  limitations  inherent  in  this  iteration  of  the  model,  it 
bears  repeating  that  these  initial  results  from  the  predictive  model  are  not  intended  to  be 
used  to  “grade”  states  and  MCCs  on  their  performance  in  promoting  effective  competition. 
The  study  team  suspects  that  the  relatively  small  amount  of  obligations  in  some  of  these 
states  and  MCCs  may  result  in  one  or  two  large  contracts  skewing  effective  competition 
rates.  Nonetheless,  the  initial  results  validate  the  ability  of  the  model  to  identify  states/MCCs 
where  rates  of  effective  competition  for  a  segment  of  the  contracting  portfolio  differ 
significantly  from  the  rates  for  the  overall  DoD.  In  addition  there  are  a  number  of  interesting 
results,  such  as  MCCs  with  a  mix  of  product  and  service  contracts  having  lower  effective 
competition  rates  and  states  with  a  focus  in  products  and  electronics  having  higher  rates. 

This  is  particularly  important  given  how  stable  rates  of  effective  competition  have 
been  for  the  DoD  overall,  and  for  DoD  products/services/R&D,  respectively.  If  gains  are  to 
be  made  in  promoting  effective  competition,  it  will  have  to  be  at  a  more  granular  level,  taking 
examples  from  states/MCCs  that  have  shown  better  than  “estimated”  rates  of  effective 
competition  and  trying  to  apply  lessons  learned  to  those  with  lower  than  ’’estimated”  rates  of 
competition.  If  nothing  else,  the  study  team  hopes  to  provide  policy-makers  with  a  road  map 
that  will  enable  them  to  identify  which  states  and  MCCs  warrant  deeper  analysis  and  to 
determine  if  their  competition  performance  is  an  artifact  of  a  few  big  contracts  skewing 
results,  or  rather  a  systemic  characteristic  that  should  either  be  studied  and  emulated  or 
studied  and  improved. 

In  an  effort  to  both  explain  a  greater  amount  of  the  noted  variance  and  to  reduce  the 
size  of  the  95%  confidence  interval  that  we  obtain,  the  CSIS  study  team  is  presently 
experimenting  with  ways  to  improve  both  the  statistical  foundations  and  the  robustness  of 
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the  predictive  model.  The  results  of  these  explorations  will  also  enable  the  study  team  to 
determine  if  additional  variables  will  be  incorporated  into  the  model.  Based  on  the  output 
from  the  current  version  of  the  model,  the  study  team  has  already  determined  to  include  a 
more  detailed  breakdown  of  services;  this  will  help  account  for  the  states  and  MCCs  that 
show  large  discrepancies  between  “estimated”  and  actual  effective  competition  rates  due  to 
higher/lower  rates  for  particular  types  of  services. 
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